{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 音乐网站用户流失预测 -- Logistic 回归\n",
    "\n",
    "### 数据集说明\n",
    "\n",
    "项目提供KKBOX用户——歌曲重复播放记录，以及用户和歌曲的元数据。训练数据由2017年2月服务到期的用户构成，target标签代表用户在2017年3月是否续订了业务。测试集中的数据由2017年3月内将到期的用户构成，需要预测用户是否在到期后的一个月内即2017年4月预定、流失的概率。\n",
    "\n",
    "以下是文件及字段说明：\n",
    "\n",
    "1. train.csv: 训练数据，共7,377,418条记录\n",
    "\n",
    "    msno: 用户id，加密String  \n",
    "\n",
    "    song_id: song id，歌曲id\n",
    "\n",
    "    source_system_tab: 触发事件的类型/tab，用于表示app的功能类型\n",
    "\n",
    "    source_screen_name: 用户看到的布局的名字（name of the layout）\n",
    "\n",
    "    source_type: 用户在app上播放音乐的入口的类型\n",
    "\n",
    "    target: 标签。1表示用户在第一次听音乐后会在一个月内继续订阅，0表示没有订阅。\n",
    "\n",
    "2. test.csv ：测试数据，共2,556,790条记录\n",
    "\n",
    "    id: id (用于结果提交)\n",
    "\n",
    "    msno: 用户id\n",
    "\n",
    "    song_id: 歌曲id\n",
    "\n",
    "    source_system_tab: 触发事件的类型/tab，用于表示app的功能类型\n",
    "\n",
    "    source_screen_name: 用户看到的布局的名字（name of the layout）\n",
    "\n",
    "    source_type: 用户在app上播放音乐的入口的类型\n",
    "\n",
    "3. sampleSubmission.csv：提交结果文件样例  \n",
    "\n",
    "    提交测试结果包含两个字段，分别为测试样本id及其标签为1的概率，格式如下：\n",
    "\n",
    "    id,target\n",
    "    \n",
    "    2,0.3\n",
    "    \n",
    "    5,0.1\n",
    "    \n",
    "    6,1\n",
    "    \n",
    "    etc.\n",
    "\n",
    "4. songs.csv：歌曲元数据信息，用unicode编码\n",
    "\n",
    "    song_id：歌曲id\n",
    "\n",
    "    song_length: 单位为ms\n",
    "\n",
    "    genre_ids: genre 类别. 可多选，用 “|“隔开\n",
    "\n",
    "    artist_name：歌手\n",
    "\n",
    "    composer：作曲\n",
    "\n",
    "    lyricist：作词\n",
    "\n",
    "    language：语言\n",
    "\n",
    "5. members.csv：用户元数据信息\n",
    "\n",
    "    msno：用户id\n",
    "\n",
    "    city：城市\n",
    "\n",
    "    bd: 年龄。注意：年龄数据有离群点\n",
    "\n",
    "    gender：性别\n",
    "\n",
    "    registered_via: 注册方式\n",
    "\n",
    "    registration_init_time: 注册时间，格式为%Y%m%d\n",
    "\n",
    "    expiration_date: 到期时间，格式为 %Y%m%d\n",
    "\n",
    "6. song_extra_infos.csv：歌曲额外的信息\n",
    "\n",
    "    song_id：歌曲id\n",
    "\n",
    "    song name ：歌曲名字\n",
    "\n",
    "    isrc – 国际标准音像制品编码(International Standard Recording Code )。理论上可用于歌曲id，但产生的ISR没有经过官方授权。因此ISRC中的信息，如国家代码和参考年份可能不正确。且多首歌曲可能共享共一个ISRC，因为一首歌曲的音像制可发行多次。\n",
    "    \n",
    "**考虑训练数据集比较大，而且特征维数比较高，先用Logistic回归模型试一下效果**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "dpath = '../data/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>msno_label</th>\n",
       "      <th>song_id_label</th>\n",
       "      <th>source_system_tab_missing</th>\n",
       "      <th>source_screen_name_missing</th>\n",
       "      <th>source_type_missing</th>\n",
       "      <th>city_1</th>\n",
       "      <th>city_3</th>\n",
       "      <th>city_4</th>\n",
       "      <th>city_5</th>\n",
       "      <th>city_6</th>\n",
       "      <th>...</th>\n",
       "      <th>language_3.0</th>\n",
       "      <th>language_10.0</th>\n",
       "      <th>language_17.0</th>\n",
       "      <th>language_24.0</th>\n",
       "      <th>language_31.0</th>\n",
       "      <th>language_38.0</th>\n",
       "      <th>language_45.0</th>\n",
       "      <th>language_52.0</th>\n",
       "      <th>language_59.0</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.265266</td>\n",
       "      <td>0.207462</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.561195</td>\n",
       "      <td>0.620835</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.561195</td>\n",
       "      <td>0.335472</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.561195</td>\n",
       "      <td>0.065859</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.265266</td>\n",
       "      <td>0.092531</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 250 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   msno_label  song_id_label  source_system_tab_missing  \\\n",
       "0    0.265266       0.207462                        0.0   \n",
       "1    0.561195       0.620835                        0.0   \n",
       "2    0.561195       0.335472                        0.0   \n",
       "3    0.561195       0.065859                        0.0   \n",
       "4    0.265266       0.092531                        0.0   \n",
       "\n",
       "   source_screen_name_missing  source_type_missing  city_1  city_3  city_4  \\\n",
       "0                         0.0                  0.0     1.0     0.0     0.0   \n",
       "1                         0.0                  0.0     0.0     0.0     0.0   \n",
       "2                         0.0                  0.0     0.0     0.0     0.0   \n",
       "3                         0.0                  0.0     0.0     0.0     0.0   \n",
       "4                         0.0                  0.0     1.0     0.0     0.0   \n",
       "\n",
       "   city_5  city_6  ...  language_3.0  language_10.0  language_17.0  \\\n",
       "0     0.0     0.0  ...           0.0            0.0            0.0   \n",
       "1     0.0     0.0  ...           0.0            0.0            0.0   \n",
       "2     0.0     0.0  ...           0.0            0.0            0.0   \n",
       "3     0.0     0.0  ...           0.0            0.0            0.0   \n",
       "4     0.0     0.0  ...           0.0            0.0            0.0   \n",
       "\n",
       "   language_24.0  language_31.0  language_38.0  language_45.0  language_52.0  \\\n",
       "0            0.0            0.0            0.0            0.0            1.0   \n",
       "1            0.0            0.0            0.0            0.0            1.0   \n",
       "2            0.0            0.0            0.0            0.0            1.0   \n",
       "3            0.0            0.0            0.0            0.0            0.0   \n",
       "4            0.0            0.0            0.0            0.0            1.0   \n",
       "\n",
       "   language_59.0  target  \n",
       "0            0.0       1  \n",
       "1            0.0       1  \n",
       "2            0.0       1  \n",
       "3            0.0       1  \n",
       "4            0.0       1  \n",
       "\n",
       "[5 rows x 250 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv(dpath + 'LR_data/Merge_Train_Scaler.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 删除ID类的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = train.drop(['msno_label','song_id_label'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['target']   \n",
    "X_train = train.drop(['target'], axis=1)\n",
    "\n",
    "# 保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "# 稀疏数据输入，模型训练会快很多\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression + GridSearchCV\n",
    "\n",
    "logistic回归的需要调整超参数有：\n",
    "\n",
    "* C（正则系数，一般在log域（取log后的值）均匀设置候选参数）\n",
    "\n",
    "* 正则函数penalty（L2/L1）\n",
    "\n",
    "目标函数为：J =  C* sum(logloss(f(xi), yi)) + penalty "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用saga（改进的随机平均梯度下降）训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13min 57s, sys: 23.7 s, total: 14min 21s\n",
      "Wall time: 1h 22min 57s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sklearn/linear_model/sag.py:334: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=None, solver='saga',\n",
       "          tol=0.0001, verbose=0, warm_start=False),\n",
       "       fit_params=None, iid='warn', n_jobs=4,\n",
       "       param_grid={'penalty': ['l2'], 'C': [0.1, 0.15, 0.2, 0.25]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penaltys = ['l2']\n",
    "Cs = [0.1, 0.15, 0.2, 0.25]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "lr_penalty= LogisticRegression(solver='saga')\n",
    "\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters, cv=5, scoring='neg_log_loss', n_jobs = 4)\n",
    "%time grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6867128587955431\n",
      "{'C': 0.1, 'penalty': 'l2'}\n"
     ]
    }
   ],
   "source": [
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**使用saga训练的最终结果：**\n",
    "\n",
    "最佳得分：0.6867128577417895\n",
    "\n",
    "最佳参数：{'C': 0.1, 'penalty': 'l2'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集上的分类报告 GridSearchCV(cv=5, error_score='raise-deprecating',\n",
      "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
      "          n_jobs=None, penalty='l2', random_state=None, solver='saga',\n",
      "          tol=0.0001, verbose=0, warm_start=False),\n",
      "       fit_params=None, iid='warn', n_jobs=4,\n",
      "       param_grid={'penalty': ['l2'], 'C': [1e-05, 0.0001, 0.001, 0.01, 0.1]},\n",
      "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
      "       scoring='neg_log_loss', verbose=0):\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.56      0.37      0.45   3662762\n",
      "           1       0.54      0.71      0.61   3714654\n",
      "\n",
      "   micro avg       0.55      0.55      0.55   7377416\n",
      "   macro avg       0.55      0.54      0.53   7377416\n",
      "weighted avg       0.55      0.55      0.53   7377416\n",
      "\n",
      "\n",
      "训练集集上的预测1的正确率：0.5450024778323467\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import accuracy_score\n",
    "y_train_pred = grid.predict(X_train)\n",
    "print(\"训练集上的分类报告 %s:\\n%s\\n\" % (grid, classification_report(y_train, y_train_pred)))\n",
    "print(\"训练集上的预测正确率：%s\\n\" % (accuracy_score(y_train, y_train_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**正确率为：0.5450024778323467，效果不好**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AUC曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "def plot_roc(labels, predict_prob):\n",
    "    false_positive_rate,true_positive_rate,thresholds=roc_curve(labels, predict_prob)\n",
    "    roc_auc=auc(false_positive_rate, true_positive_rate)\n",
    "    plt.title('ROC')\n",
    "    plt.plot(false_positive_rate, true_positive_rate,'b',label='AUC = %0.4f'% roc_auc)\n",
    "    plt.legend(loc='lower right')\n",
    "    plt.plot([0,1],[0,1],'r--')\n",
    "    plt.ylabel('TPR')\n",
    "    plt.xlabel('FPR')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUZfbH8c+R6io2sNEEBVdpixgLqOiKBbHhWhY7LMoqKwI2cMUCq4JiV4qIDQuI/FbFFcVGs4C0iBSVSC9KQGkiJeH5/XEGjJCQhMzNZDLf9+vFy5k7d+6cKzpnnnYeCyEgIiKpa49EByAiIomlRCAikuKUCEREUpwSgYhIilMiEBFJcUoEIiIpTolARCTFKRGI7IKZLTCz38xsvZn9aGYvmdneOV5vZmafmtk6M1tjZu+aWb0drrGPmT1hZoti1/kh9rxK8d+RyM6UCETyd34IYW+gMXAMcCeAmTUFPgTeAaoCtYGvgc/N7PDYOeWBT4D6QEtgH6ApsAo4vnhvQyR3ppXFInkzswXAdSGEj2PPHwbqhxDONbMJwDchhI47vOd9IDOEcI2ZXQc8ABwRQlhfzOGLFIhaBCIFZGbVgXOADDP7E9AMeDOXU4cDZ8YenwF8oCQgJZkSgUj+3jazdcBiYAVwL3AA/v/P8lzOXw5s6/+vnMc5IiWGEoFI/lqHECoBpwFH4V/yvwBbgUNzOf9QYGXs8ao8zhEpMZQIRAoohDAOeAl4JITwK/AlcGkup16GDxADfAycbWZ7FUuQIrtBiUCkcJ4AzjSzvwDdgWvN7GYzq2Rm+5vZ/fisoJ6x81/Bu5T+z8yOMrM9zKyymf3bzFol5hZE/kiJQKQQQgiZwBDgnhDCZ8DZwN/wcYCF+PTSk0MIc2Pnb8IHjL8FPgLWAl/h3UuTiv0GRHKh6aMiIilOLQIRkRSnRCAikuKUCEREUpwSgYhIiiub6AAKq0qVKqFWrVqJDkNEJKlMnTp1ZQjhwNxeS7pEUKtWLaZMmZLoMEREkoqZLczrNXUNiYikOCUCEZEUp0QgIpLilAhERFKcEoGISIqLLBGY2QtmtsLMZubxupnZU2aWYWYzzKxJVLGIiEjeomwRvIRv1p2Xc4C6sT8dgAERxiIiInmILBGEEMYDP+/ilAuBIcFNBPYzM+3kJCKyg1WLfqXvvxaQkRHN9RM5RlAN37BjmyWxYzsxsw5mNsXMpmRmZhZLcCIiibZyJTx3+aesqdWIFv3/xocfbI3kc5JisDiEMCiEkBZCSDvwwFxXSIuIlBqZmXBfl9W8e+j1XD+sBXvutQf7v/g4HW+K5is7kSUmlgI1cjyvHjsmIpKSVqyARx6Bgf2ymbihGUfZd6z8xx0c+sx9sOeekX1uIhPBSOAmMxsGnACsCSEsT2A8IiIJsWIF9O0Lw/qtYtnGA2hzeRn2OfEB9mhagyppaZF/fmSJwMyGAqcBVcxsCXAvUA4ghDAQGAW0AjKADUC7qGIRESmJfvzRE8CA/oFLNr3G7HKd+bVnHw65+3rgomKLI7JEEEK4PJ/XA/CvqD5fRKSk+vFHePhhGDgQDty4mC+q3kDjpaOgyYlUuvikYo8nKQaLRURKg+XLoUsXqF0bnnoKHk0byrw/1afxL2PhiSfgs8+gXr1ijyvp9iMQEUk2S5fCQw/BoEGQlQXXXAP//jfUydgfHj3BX6hdO2HxKRGIiERk6VLo0weee84TQLurs+hz8ONUrrQZ6twFdVrC2WeDWULjVNeQiEicLV4M//oXHH64jwNcfTUsHPk1z31zIpUfugNmzIAQ/OQEJwFQIhARiZvFi6FjR6hTx3t7rr0W5s7cxHOH3E21C9P8hDffhGHDSkQC2EaJQESkiBYtghtvhCOOgMGDoV07yMjwZFBry1wfILjiCpg9Gy65pEQlAdAYgYjIblu4EB58EF580Z+3bw933gk1D1gP77wDh10JDRrAt996P1EJpRaBiEghLVgAHTp4F9BLL8H118MPP8CAAVDzu4+gYUMfGJgzx99QgpMAKBGIiBTYvHlw3XVQty68/DL885+eAPr1gxp7/+JNgrPOgvLlYdw4OProRIdcIOoaEhHJx7x58MAD/uVftqyPB3TrBtW2Fc7PzoaTToLvv/e+oXvugYoVExpzYSgRiIjkISPDE8Arr0C5cj4ltFs3qFo1dsLKlXDAAVCmjA8W1KwJTZJv1111DYmI7GDuXJ/6edRRPtOzUydvFTz5ZCwJhABDhsCRR/o0IYDWrZMyCYBaBCIi233/Pdx/P7z2GlSoADffDHfcAYcckuOkhQt9cGD0aGjWDJo3T1i88aJEICIp79tvPQEMHeoJoGtXuP12OPjgHU589VUfIAgBnn7aV4/tkfwdK0oEIpKy5sz5PQHsuSfccgvcdlsuCWCbAw/0QeFnn4XDDivWWKOkRCAiKWf2bPjPf+CNN+BPf/Jf/7feCgcdtMOJW7bAo4/6P+++2wvEnXVWiVsZXFRKBCKSMmbN8gQwfLgngDvu8ARw4IG5nDx9uq8LmD4d2rTx7iCzUpcEQLOGRCQFzJwJl13mC37few+6d/fVwX365JIENm70zQKOOw6WLYP/+z/vOyqFCWAbtQhEpNT65hvo1QtGjIBKlfz7vWtXqFx5F2/KyIBHHvHdYx59FPbfv9jiTRQlAhEpdb7+2hPAf/8L++wDPXp4AjjggDzesH49vPWW1wdq0AC++y6hO4YVN3UNiUipMX06XHQRNG4MH3/slR4WLPBxgTyTwOjRUL++ryDbViQuhZIAKBGISCkwbdrvC3vHjIF77/UE0LPnLnp2Vq3yL/+WLX3keMKEpCkSF2/qGhKRpDV1qn/Zv/su7LefP775Zn+8S9uKxGVkwF13ed9REhWJizclAhFJOlOm+Jf+//7nv/h79fIEsO+++bwxM9NHisuU8V3DDjvM+5FSnLqGRCRpfPUVnHuuz+z84gtfFbxgga/12mUSCMG3ETvySHjuOT924YVKAjFKBCJS4k2aBK1awQknwMSJXhp6/nzv1dlnn3zevGCBrwj+xz98IcFf/1ocIScVdQ2JSIn15ZfeBTR6tPfo9O7tewJUqlTAC7zyiheJM4P+/b1qaCkoEhdvSgQiUuJ88YUngA8/hCpVvDu/Y0fYe+9CXujgg71M9MCBvmmM5EqJQERKjM8+8wTw8cde+uHhh/0HfYETwJYt/qbsbF9EcNZZ/kd2SW0kEUm4CRPgjDPglFNgxgyv8DB/vlcFLXASmDbNR5F79PCVwSFEGnNpokQgIgkzbhycfrr33syc6aV95s/3iqB77VXAi/z2m1eRO/54+OknLxXx2mulukhcvEWaCMyspZl9Z2YZZtY9l9drmtkYM5tuZjPMrFWU8YhIyTB2LJx2mv+ZMwcef9z3BL7lFl/kWyjz5sFjj0Hbtr7RQOvWcY+3tIssEZhZGaAfcA5QD7jczOrtcFoPYHgI4RigDdA/qnhEJLFCgE8/hVNP9Rmc338PTzzh3+NduhQyAaxdCy+95I/r1/fd5gcPTolKoVGIskVwPJARQpgXQtgMDAMu3OGcAGybBbwvsCzCeEQkAUKATz7x7p8WLbyqw1NPwQ8/QOfOvkVkoYwa5RVC27f/vUhcKdo2MhGiTATVgMU5ni+JHcvpPuAqM1sCjAI65XYhM+tgZlPMbEpmZmYUsYpInIUAH33kA8BnnOF9/08/7QmgU6fdSAArV3qZ6HPP9YUEn3+eskXi4i3Rg8WXAy+FEKoDrYBXzGynmEIIg0IIaSGEtANz3VNOREqKEHz+/0kn+czNhQuhXz9PADfdtJu13bYViRs2zKeFTpsGJ54Y99hTVZTrCJYCNXI8rx47llN7oCVACOFLM6sIVAFWRBiXiEQgBF8B3LOnl4GoUQMGDIB27aBChd286E8/+YKCMmV8Tulhh0GjRnGNW6JtEUwG6ppZbTMrjw8Gj9zhnEVACwAzOxqoCKjvRySJhADvvw9Nm8I55/g2vwMH+vjtDTfsZhIIAZ5/Hv78Zxg0yI+df76SQEQiSwQhhCzgJmA0MAefHTTLzHqZ2QWx024Frjezr4GhQNsQtApEJBmE4BvBn3CCF4T78Uf/zp4710v67HYrYN48H1S47jqvDnrGGXGNW3YWaYmJEMIofBA457F7cjyeDZwUZQwiEl/bEkDPnr4vQK1aXtn5mmugfPkiXvzll72oUJky3qy4/noViSsG+jcsIgUSAowc6VUczj/fd3p8/nlfD3DddXFIAgBVq/pS49mzVSm0GKnonIjsUgjwzju+C9j06XDEEfDCC3DVVVCuXBEvvnkz9OkDW7fCfffBmWf6HylWSrcikqutW71szzHHwEUXwbp1vpj32299JlCRk8DkyXDssb7T/Lx5KhKXQEoEIvIHW7fC//2fJ4C//Q02bIAhQ3wR77XXQtmi9iNs2AC33ebrAH75xfubhgxRkbgEUiIQEcATwIgRPlHnkktg40bf4Gv2bF/QW+QEsM22JcbXXw+zZvmAgySUEoFIitu6FYYPh7/8BS691LvtX3vNE8BVV8UpAaxZ45vHgxeJy8jwWUG73HFeiosSgUiKys6GN97w/dz//nd//vrr/iP9iit8BmdcvPeef/lfd50PMIAvO5YSQ4lAJMVkZ8PQoZ4A2rTxY8OGwTffwOWXxzEBZGbClVfCeed5eegvv4SjjorTxSWeNH1UJEVsawH85z/+w7x+fX9+ySURTNfPzoaTT/bxgJ49fQexuCw0kCgoEYiUcllZ/ov//vt9K98GDeDNN31GUNwTwI8/wkEHebPi0Ud92XGDBnH+EIk3dQ2JlGI//ODF4K6+2mv/jBgBX38dQStg61Z49lk48kj/J3iXkJJAUlCLQKSUeuMNn6FZpowPAv/97xFVbMjI8A8aO9bLQ5x9dgQfIlFSi0CklPntNy/T06aNjwOkp/sgcCRJ4MUXfdR52jSvPPfxx3D44RF8kERJiUCkFJkzB44/3stBd+sG48dHvJ1vzZreApg926eHanVwUlLXkEgpEIJXcP7Xv2CvvXyjmJYtI/igTZugd28fE+jVy3ejb9Eigg+S4qQWgUiSW7fO9wJo185bA+npESWBSZO8SFzPnrBokYrElSJKBCJJLD0d0tJ8MLhnT++ir1o1zh/y669wyy0+/WjNGvjf/7wMqbqBSg0lApEkFAL06+cFPNevh08/hXvuieOq4JwWLoT+/X0D4lmz4NxzI/gQSSSNEYgkmdWroX17+O9/fbP4l1+GAw+M4ENGjPAB4Hr1fIpo9epx/hApKdQiEEkikyb5PgEjR0Lfvt5LE/ck8M47/uV/ww2/F4lTEijVlAhEksDWrfDII16+JwT47DPf2yWuawNWrPDFB61be3aZOFFF4lKEuoZESrjMTN8Z7P33vT7Q4MFezDOusrPhpJN8NtD998Mdd8RhL0pJFkoEIiXYuHG+N8DKlT44fOONcZ6ss2wZHHKIjzI/+aQXiatXL44fIMlAXUMiJVB2tk8HPf102HtvHxvo2DGOSWDrVhgwwLt+Bg70Y61aKQmkKLUIREqYZct8i8gxY/yf/ftDpUpx/IDvv/cicePHwxln+NQjSWlqEYiUIKNH++bxkyZ5PbchQ+KcBJ5/3jcnnjEDXngBPvwQateO4wdIMlIiECkBtmzxTbxatoSDD4YpU6Bt2wgW79aq5S2A2bO9JoVWBwvqGhJJuIULvUz0l196+ejHH4c994zTxTdt8r0pwWcDqUic5EItApEEevtt7wqaOdO3kxw4MI5J4Isv/OIPPADLl6tInORJiUAkATZuhE6d4KKL4IgjYPp030EsLtavh86dffXZhg3wwQc+NqBuIMlDpInAzFqa2XdmlmFm3fM45zIzm21ms8zs9SjjESkJ5s6FZs3gmWegSxf4/HNPBnGzaJHvG/yvf3lTQ1tHSj4iGyMwszJAP+BMYAkw2cxGhhBm5zinLnAncFII4RczOyiqeERKgtdf93GA8uW9XtD558fpwr/8Am++CR06+FqAefMiqEctpVWULYLjgYwQwrwQwmZgGHDhDudcD/QLIfwCEEJYEWE8IgmzYYMX8rzySp+9mZ4exyTw1lv+5d+xI3z3nR9TEpBCiDIRVAMW53i+JHYspyOBI83sczObaGa57qtkZh3MbIqZTcnMzIwoXJFozJoFxx3n0/bvugvGjoUaNeJw4R9/hEsv9QJEhxwCX30Ff/5zHC4sqSbR00fLAnWB04DqwHgzaxhCWJ3zpBDCIGAQQFpamqY+SFIIwcdob77ZF4WNHg1nnhmni2dnwymnwOLF8OCDXopUReJkN0WZCJYCOX/3VI8dy2kJMCmEsAWYb2bf44lhcoRxiURu7VofCxg2zKftv/qq/2gvsiVLvNunTBl46ilfFaxS0VJEUXYNTQbqmlltMysPtAFG7nDO23hrADOrgncVzYswJpHITZ0KTZrA8OG+hmv06Dgkga1b4emn/Ut/wAA/ds45SgISF5ElghBCFnATMBqYAwwPIcwys15mdkHstNHAKjObDYwBbg8hrIoqJpEoheA/0ps29QW948b5mECR9xH+9lto3tz7mE4+Gc47Ly7ximxjIclWG6alpYUpU6YkOgyRP/j5Z99H+O23/Xv6pZegcuU4XHjwYLjpJvjTn+CJJ+Dqq7UwTHaLmU0NIaTl9ppWFosU0Rdf+D7C773ndYJGjoxTEgBfaXb++TBnDlxzjZKARCLRs4ZEktbWrfDww9CjB9Ss6SuEjzuuiBfduBF69fLHDz4If/2r/xGJkFoEIrvhp598rPbOO+Hii71WUJGTwOefe5G43r19o+Ik67aV5KVEIFJIn37q39fjx3u10GHDYN99i3DBdeu8At0pp/go8+jR8Nxz6gaSYqNEIFJAWVlwzz2+u+N++/lC3n/+Mw7f10uW+KBwp07wzTdw1llxiVekoDRGIFIAS5fCFVd4K6BtW68cutdeRbjgqlW+0ODGG+Hoo71I3KGHxitckUIpdIvAzPYwsyujCEakJHrvPS8UN3Wq7yH84otFSAIhwIgRXiTu5pt/LxKnJCAJlGciMLN9zOxOM3vGzM4y1wlf+XtZ8YUokhibN3sJn/POg+rVPRFcfXURLrh8uY8sX3qpV52bMkVF4qRE2FXX0CvAL8CXwHXAvwEDWocQ0oshNpGEmT8f2rTxcYCOHeHRR6FixSJccFuRuKVLfc5p165QVj2zUjLs6r/Ew0MIDQHMbDCwHKgZQthYLJGJJMiIEb53wLbHF19chIstXgzVqnmdiX79vEjckUfGJU6ReNnVGMGWbQ9CCNnAEiUBKc02bvRf/5de6j0206cXIQlkZ3vhoZxF4s4+W0lASqRdtQj+YmZr8e4ggD1zPA8hhH0ij06kmHz3nW8e//XXPi7wwAO+neRumTPHCw99+aWvOovbVmQi0cgzEYQQilozUSQpvPKKz+KsWNFnCLVqVYSLDRrk6wEqVfILX3mlFoZJiZdnIjCzisANQB1gBvBCrLS0SKmwfr0X9nz5Za/y/Prr3p1fJHXrwkUXebfQQQfFJU6RqO2qa+hlfJxgAtAKqA90Lo6gRKI2Y4Z3BX33na8Wvvvu3ZzE89tvcN99/qu/Tx8ViZOktKv/9OvlmDX0PPBV8YQkEp0QvPemc2fYf3/4+GM4/fTdvNj48T69aO5cuOEGv7i6gSQJFXTWkLqEJOmtWeNrA264AU491QeGdysJrF3r04tOPdVnB33yic8MUhKQJLWrFkHj2Cwh8JlCmjUkSWvyZE8CCxd6D87tt8Meu1tycdky34Lsllt874AiFR0SSbxdJYKvQwjHFFskIhEIwXd47NbNy/mMHw/Nmu3GhVau9CJxHTv62oD58+Hgg+Mer0gi7Oo3kXbFkKS2ahVccIH/cD/3XF8gVugkEAK88YYXievSBb7/3o8rCUgpsqsWwUFmdkteL4YQHosgHpG4mDDBy0avWOEzOW+6aTe68Jct8wUGI0dCWpqPBWhlsJRCu0oEZYC9+X1lsUiJl53tYwD33AOHH+6Le5s02c0LNW/uReIeecSnGalInJRSu/ove3kIoVexRSJSRD/+6GWiP/4YLr/ct5Hcp7BTGhYu9JrTZcpA//6eTerUiSRekZJiV2MEaglI0vjoI9885vPPfdfH114rZBLIzobHHvPdwrYViTvrLCUBSQm7SgQtii0Kkd2UlQV33eWFPatU8Wmi7dsXcjxg5kwfRb71VmjRAlq3jixekZIoz0QQQvi5OAMRKazFi+G00+DBB/3Lf/JkqF+/kBcZONAHEebN82JDI0d615BICtHolySlkSOhXTvfTvK113yGUKFsKwdx9NG+AcETT8CBB0YSq0hJt7trK0USYvNm3+XxwgvhsMNg2rRCJoENG3zDge7d/fmpp3omURKQFKZEIEnjhx/gpJP8x3unTj41tG7dQlxg7Fho1Mg3IF6/3lsFIqJEIMnhjTfgmGMgIwPeessXiVWoUMA3r1kD//zn7+WhP/3U9w9WkTgRQIlASrjffvPv8DZtoEEDSE/fjUk9y5fDq696l9CMGdovQGQHkSYCM2tpZt+ZWYaZdd/FeRebWTCztCjjkeQyezYcf7zvH9C9O4wb5+MCBZKZCU8/7Y+POgoWLIC+feFPf4oqXJGkFVkiMLMyQD/gHKAecLmZ1cvlvEr4zmeToopFkksI8OKLcNxx8NNP8MEH0Ls3lCtXwDe//rrPBrr11t+LxGkwWCRPUbYIjgcyQgjzQgibgWHAhbmc9x/gIWBjhLFIkli3Dq65Bv7xDzjhBO8KOvvsAr558WI4/3zfML5OHS83qiJxIvmKMhFUAxbneL4kdmw7M2sC1AghvLerC5lZBzObYmZTMjMz4x+plAjp6V7k8/XXoWdPLxtRtWoB35yV5avLxoyBxx/3WhOFXl0mkpoStqDMzPYAHgPa5nduCGEQMAggLS1Nc/5KmRC8vtutt0Llyj6p59RTC/jmBQugRg2vDPrss14k7vDDowxXpNSJskWwFKiR43n12LFtKgENgLFmtgA4ERipAePUsno1XHKJ7xfQooW3CgqUBLKyvDz00Ud7FgE44wwlAZHdEGWLYDJQ18xq4wmgDbB9DWgIYQ1QZdtzMxsL3BZCmBJhTFKCTJzo00K3lfzv2rWA+wjPmOHFhaZM8SXGF18ceawipVlkLYIQQhZwEzAamAMMDyHMMrNeZnZBVJ8rJd/WrT6T85RTfE3XZ595t1CBkkD//nDssb5vwBtv+OqyAg8kiEhuIh0jCCGMAkbtcOyePM49LcpYpGTIzIRrr4X33/cf8oMHw377FeCN24rENWjgzYjHH/e60yJSZKo+KsVm3DgvELdqlVd4uPHGAlR5+PVX6NHDB4P79vXtI5s3L5Z4RVKFSkxI5LKzfTro6afD3nv72EDHjgVIAp98Ag0bepW5TZtUJE4kIkoEEqlly+DMM+G++3yd19Sp0LhxPm9avRquu85nAZUtC+PHe5U5FYkTiYQSgUTmgw/8S3/SJHjpJRgyxFsE+frpJxg2DLp1g6+/9lFlEYmMEoHE3ZYt/h1+zjlwyCE+y/Paa/N5008/wZNP+uM//9kXivXpA3vuGXW4IilPiUDiauFCH8t9+GEvHz1pkq/5ylMIXiK6Xj244w6YO9ePa0aQSLFRIpC4eftt7wqaPdun+A8cmM8P+kWL4Nxz4eqrvRWQnl7ILcdEJB6UCKTINm70rSMvuuj3op+XXZbPm7YVids2EDxhQj5NBxGJitYRSJHMnQt//7t/+Xft6t365cvv4g3z5vnuMmXLwnPPwRFHQK1axRWuiORCLQLZba+/Dk2a+LjAyJHw2GO7SAJZWfDQQz4W0K+fH2vRQklApARQIpBC27DBp/lfeaWPCaSn+34weUpP911muneHVq3g0kuLLVYRyZ8SgRTKrFm+heQLL8Bdd/k+MDVq7OINzzzjb1i6FEaMgP/+Fw49tNjiFZH8aYxACiQEeP55uPlm2Gcf+PBDX/i7yzeYQaNG3nR47DE44IBii1dECk6JQPK1dq2vCRg2zL/8X3nFF4rlav16byqUK+ebDKhInEiJp64h2aWpU31A+M034YEHYPToXSSBDz/0MtFPP+3Li1UkTiQpKBFIrkLw6f1Nm3rhz7Fj4d//zmPzmF9+gXbt4OyzoWJFXxvw5JMqEieSJJQIZCc//+yLwzp3hpYtfdLPySfv4g0rVvhA8J13FuBkESlpNEYgf/DFF3D55bB8uW8C1rlzHj/sf/wRhg71VWTbisRVrlzc4YpIHKhFIIDvI9ynj4/rli3rCaFLl1ySQAjw8su+MOzOO38vEqckIJK0lAiEn37yktF33un7CE+bBmlpuZy4YIH3FbVt64lAReJESgV1DaW4Tz/1af6rV8Ozz8L11+fRFZSVBX/9K6xc6SUibrghj5FjEUk2SgQpKisLevWC++/3Lv4PP/TtgXeSkQG1a3t/0QsvwOGHe9E4ESk19JMuBS1d6vXe/vMf7+WZMiWXJLBlCzz4INSv/3uRuL/+VUlApBRSiyDFvPeebxu5caOvEL7qqlxOmjYN2rf3MYBLL/U60yJSaqlFkCI2b4bbboPzzoPq1f27Ptck8NRTcPzxPj30v/+F4cPh4IOLPV4RKT5KBClg/nw45RR49FHo2BEmToQjj9zhpG3lII45Bq65xvebvOiiYo9VRIqfuoZKuREjfO+AbY8vvniHE9at83mjFSp4pjjlFP8jIilDLYJSauNG//V/6aVw1FG+leROSeCDD7xIXP/+3iJQkTiRlKREUAp9951vCDZgANx+u+8LX7t2jhNWrfIR43POgb32gs8/9/0CVCROJCUpEZQyr7wCxx4Ly5bBqFHw8MO+NcAfrFoFb70Fd9/tTYWmTRMSq4iUDJEmAjNraWbfmVmGmXXP5fVbzGy2mc0ws0/MTJPUd9P69b4m4JprPBGkp/sP/u2WL/eNYkLwkeKFC31FWYUKiQpZREqIyBKBmZUB+gHnAPWAy82s3g6nTQfSQgiNgBHAw1HFU5rNmOHbAg8ZAvfcA598AtWqxV4MwVcEH320twAyMvz4/vsnLF4RKVmibBEcD2SEEOaFEA+dEKkAABG5SURBVDYDw4ALc54QQhgTQtgQezoRqB5hPKVOCDBwoE/7X7PGE0DPnl4NAvB5o2ed5YvD/vIX+PprFYkTkZ1EOX20GrA4x/MlwAm7OL898H5uL5hZB6ADQM2aNeMVX1Jbs8YLxL35pm8MNmQIHHRQjhOysuD00308YMAA6NBBReJEJFclYh2BmV0FpAGn5vZ6CGEQMAggLS0t5ec4Tp7sVR8WLYKHHvIVw9u/4+fO9cJwZcvCiy/CEUdAjRoJjVdESrYofyIuBXJ+A1WPHfsDMzsDuAu4IISwKcJ4kl4IvmvYSSdBdrZPC73jjlgS2LLFS4k2aADPPONvOO00JQERyVeULYLJQF0zq40ngDbAFTlPMLNjgGeBliGEFRHGkvRWrfJZQf/7H7Ru7eO/28d7p0zxcYAZM6BNG99rUkSkgCJrEYQQsoCbgNHAHGB4CGGWmfUyswtip/UF9gbeNLN0MxsZVTzJbMIEaNzY9wx46imvBbc9CTz5pK8eW7kS3nnH9xH+w2CBiMiuRTpGEEIYBYza4dg9OR6fEeXnJ7vsbOjdG+6917v9v/wSmjSJvRiCrwROS/PWwMMPw377JTReEUlOJWKwWHb2449eJvqTT+CKK3yaaKVKwNq10K0bVKz4+4DBSSclOlwRSWKaT1gCffSRT/v/4gt4/nl49dVYEhg1yncMGzTIZwWpSJyIxIESQQmSlQV33eXrAg480KeJ/uMfYKtWevPg3HNh3309Q/TtqyJxIhIXSgQlxOLFPtvzwQd9/4CvvvIf/wD88gu8+64PFkyb5oPDIiJxojGCEmDkSGjXzreTfP312OzPpUvhmde8jnTdul4kToPBIhIBtQgSaNMm6NIFLrwQatXyitCXtwnw3HNQrx7cdx/88IOfrCQgIhFRIkiQH37wyT5PPgk33+zd/nXsB2jRwusCNWniC8Tq1El0qCJSyqlrKAHeeMMLxpUt6/vDtG6NjxS3aAE//wzPPusDBSoSJyLFQImgGP32m3cFDRoEzZr5IuCav30HWUd4Vnj5ZS8SV13VuEWk+OgnZzGZPdv3DRg0CO68E8Z+uJmaL/aEhg2hXz8/6dRTlQREpNipRRCxEOCll+Cmm3yf+A8+gLP3/wpObA8zZ/qy4SuvTHSYIpLC1CKI0Lp1cPXVvijsxBN9g7Cz5zzhm8VvWxvw2mtQpUqiQxWRFKZEEJH0dN9EfuhQ3yP+w9GBQw/F+4euvx5mzYLzzkt0mCIiSgTxFoJ3+Z94Ivz6K4x/dw13L/knZW7r6ic0a+YV5PbdN7GBiojEKBHE0erVcMklPh5wxhkw5+F3Oen6ejB4MFSooCJxIlIiKRHEycSJvnnMyJHQv2cm71a6gn2uugAqV/YXH3pIReJEpERSIiiirVu9EOgpp/j3/Oefw41XrMHeHwU9e/o2kscdl+gwRUTypOmjRZCZCddeC++/D/9stZjHj32VPY/rDlbHi8RpHEBEkoBaBLtp3DjvChrzyVbGthnIgAn12fPR+38vEqckICJJQomgkLKzvcfn9NOhfvm5ZDY8nVOH3Ygdfzx8842KxIlI0lHXUCEsW+YbhY0ZA22vyuL58WeyR8Zq30+yXTsNBotIUlIiKKAPPoBrroHq6+bw8vN1ueYfZWHCK14krmrVRIcnUiy2bNnCkiVL2LhxY6JDkTxUrFiR6tWrU65cuQK/R4kgH1u2QI8e8MTDm3jyoAf5Z9aD2Nq+QBefKiSSQpYsWUKlSpWoVasWphZwiRNCYNWqVSxZsoTatWsX+H1KBLuwYIFvGxkmTmT+/u2pumK2Fw+6+upEhyaSEBs3blQSKMHMjMqVK5OZmVmo92mwOA9vvQXHHAOnT3+UL60ZVfdeB6NGwZAhvkhMJEUpCZRsu/P3o0Swg40boVMnuPhvW6lTBzq+0hS74QYvGX3OOYkOT0Qk7pQIcpg7F846fjWNn2nP+Mad+fxzqHZpM+jfH/bZJ9HhiUjM22+/jZnx7bffbj82duxYztuhom/btm0ZMWIE4APd3bt3p27dujRp0oSmTZvy/vvvFzmW3r17U6dOHf785z8zevToXM9p27YttWvXpnHjxjRu3Jj09PQ/vD558mTKli27PdaFCxfSpEkTGjduTP369Rk4cOD2c4cOHUrDhg1p1KgRLVu2ZOXKlUW+ByWCmNdfh3savc3wmfVot8fLnHxOJcqXU5E4kZJo6NChnHzyyQwdOrTA77n77rtZvnw5M2fOZNq0abz99tusW7euSHHMnj2bYcOGMWvWLD744AM6duxIdnZ2ruf27duX9PR00tPTady48fbj2dnZdOvWjbPOOmv7sUMPPZQvv/yS9PR0Jk2aRJ8+fVi2bBlZWVl07tyZMWPGMGPGDBo1asQzzzxTpHsADRbz669w1/UraDb0JobyJpvrN2aPIf+DJk0SHZpIidali++7EU+NG8MTT+z6nPXr1/PZZ58xZswYzj//fHr27JnvdTds2MBzzz3H/PnzqVChAgAHH3wwl112WZHifeedd2jTpg0VKlSgdu3a1KlTh6+++oqmTZsW+BpPP/00F198MZMnT95+rHz58tsfb9q0ia1btwI+KyiEwK+//krlypVZu3YtdeKwiDWlWwQzZ/o+Me8NXcv5FT8i+z8PUH76V0oCIiXYO++8Q8uWLTnyyCOpXLkyU6dOzfc9GRkZ1KxZk30K0MXbtWvX7V04Of/06dNnp3OXLl1KjRo1tj+vXr06S5cuzfW6d911F40aNaJr165s2rRp+/vfeustbrzxxp3OX7x4MY0aNaJGjRp069aNqlWrUq5cOQYMGEDDhg2pWrUqs2fPpn379vneU35SskUQAgx7eBHf9niFVQf8m1c/qsOeJyyCSpUSHZpI0sjvl3tUhg4dSufOnQFo06YNQ4cO5dhjj81ztkxhZ9E8/vjjRY5xR7179+aQQw5h8+bNdOjQgYceeoh77rmHLl268NBDD7HHHjv/Jq9RowYzZsxg2bJltG7dmksuuYQDDjiAAQMGMH36dA4//HA6depE79696dGjR5HiizQRmFlL4EmgDDA4hNBnh9crAEOAY4FVwN9DCAuijGnt6q2MOGMgl07txkV7bKXTO3+nyol1ACUBkZLu559/5tNPP+Wbb77BzMjOzsbM6Nu3L5UrV+aXX37Z6fwqVapQp04dFi1axNq1a/NtFXTt2pUxY8bsdLxNmzZ07979D8eqVavG4sWLtz9fsmQJ1apV2+m9hx56KAAVKlSgXbt2PPLIIwBMmTKFNm3aALBy5UpGjRpF2bJlad269fb3Vq1alQYNGjBhwgQOO+wwAI444ggALrvsslxbKoW2rc8p3n/wL/8fgMOB8sDXQL0dzukIDIw9bgO8kd91jz322LC7vhnxbZhU8ZQQIMyrc2bI/mH+bl9LJBXNnj07oZ//7LPPhg4dOvzhWPPmzcO4cePCxo0bQ61atbbHuGDBglCzZs2wevXqEEIIt99+e2jbtm3YtGlTCCGEFStWhOHDhxcpnpkzZ4ZGjRqFjRs3hnnz5oXatWuHrKysnc5btmxZCCGErVu3hs6dO4du3brtdM61114b3nzzzRBCCIsXLw4bNmwIIYTw888/h7p164YZM2aEpUuXhkMOOSSsWLEihBBCjx49wi233LLTtXL7ewKmhDy+V6NsERwPZIQQ5gGY2TDgQmB2jnMuBO6LPR4BPGNmFgs6roa8kEXz9mdTw9Yw998vUvf+a1UkTiTJDB06lG7duv3h2MUXX8zQoUNp3rw5r776Ku3atWPjxo2UK1eOwYMHs2+sJPz9999Pjx49qFevHhUrVmSvvfaiV69eRYqnfv36XHbZZdSrV4+yZcvSr18/ypQpA0CrVq0YPHgwVatW5corryQzM5MQAo0bN/7DdNDczJkzh1tvvRUzI4TAbbfdRsOGDQG49957ad68OeXKleOwww7jpZdeKtI9AFgE37l+YbNLgJYhhOtiz68GTggh3JTjnJmxc5bEnv8QO2flDtfqAHQAqFmz5rELFy4sdDxffAHv3fkZt/Y/ggPqH7q7tyWS0ubMmcPRRx+d6DAkH7n9PZnZ1BBCWm7nJ8VgcQhhEDAIIC0tbbcyV7Nm0GzcyXGNS0SkNIhy+uhSoEaO59Vjx3I9x8zKAvvig8YiIlJMokwEk4G6ZlbbzMrjg8EjdzhnJHBt7PElwKdRjA+ISPzof9GSbXf+fiJLBCGELOAmYDQwBxgeQphlZr3M7ILYac8Dlc0sA7gF6J771USkJKhYsSKrVq1SMiihQmw/gooVKxbqfZENFkclLS0tTJkyJdFhiKQk7VBW8uW1Q1nSDxaLSMlQrly5Qu18JckhpWsNiYiIEoGISMpTIhARSXFJN1hsZplA4ZcWuypA0bfzSS6659Sge04NRbnnw0IIB+b2QtIlgqIwsyl5jZqXVrrn1KB7Tg1R3bO6hkREUpwSgYhIiku1RDAo0QEkgO45NeieU0Mk95xSYwQiIrKzVGsRiIjIDpQIRERSXKlMBGbW0sy+M7MMM9upoqmZVTCzN2KvTzKzWsUfZXwV4J5vMbPZZjbDzD4xs8MSEWc85XfPOc672MyCmSX9VMOC3LOZXRb7u55lZq8Xd4zxVoD/tmua2Rgzmx7777tVIuKMFzN7wcxWxHZwzO11M7OnYv8+ZphZkyJ/aF6bGSfrH6AM8ANwOFAe+Bqot8M5HYGBscdtgDcSHXcx3PNfgT/FHt+YCvccO68SMB6YCKQlOu5i+HuuC0wH9o89PyjRcRfDPQ8Cbow9rgcsSHTcRbzn5kATYGYer7cC3gcMOBGYVNTPLI0tguOBjBDCvBDCZmAYcOEO51wIvBx7PAJoYZbUO9nne88hhDEhhA2xpxPxHeOSWUH+ngH+AzwElIa6yQW55+uBfiGEXwBCCCuKOcZ4K8g9B2Cf2ON9gWXFGF/chRDGAz/v4pQLgSHBTQT2M7MibcReGhNBNWBxjudLYsdyPSf4BjprgMrFEl00CnLPObXHf1Eks3zvOdZkrhFCeK84A4tQQf6ejwSONLPPzWyimbUstuiiUZB7vg+4ysyWAKOATsUTWsIU9v/3fGk/ghRjZlcBacCpiY4lSma2B/AY0DbBoRS3snj30Gl4q2+8mTUMIaxOaFTRuhx4KYTwqJk1BV4xswYhhK2JDixZlMYWwVKgRo7n1WPHcj3HzMrizclVxRJdNApyz5jZGcBdwAUhhE3FFFtU8rvnSkADYKyZLcD7Ukcm+YBxQf6elwAjQwhbQgjzge/xxJCsCnLP7YHhACGEL4GKeHG20qpA/78XRmlMBJOBumZW28zK44PBI3c4ZyRwbezxJcCnITYKk6TyvWczOwZ4Fk8Cyd5vDPnccwhhTQihSgihVgihFj4uckEIIZn3OS3If9tv460BzKwK3lU0rziDjLOC3PMioAWAmR2NJ4LMYo2yeI0EronNHjoRWBNCWF6UC5a6rqEQQpaZ3QSMxmccvBBCmGVmvYApIYSRwPN48zEDH5Rpk7iIi66A99wX2Bt4MzYuviiEcEHCgi6iAt5zqVLAex4NnGVms4Fs4PYQQtK2dgt4z7cCz5lZV3zguG0y/7Azs6F4Mq8SG/e4FygHEEIYiI+DtAIygA1AuyJ/ZhL/+xIRkTgojV1DIiJSCEoEIiIpTolARCTFKRGIiKQ4JQIRkRSnRCBSQGaWbWbpOf7UMrPTzGxN7PkcM7s3dm7O49+a2SOJjl8kL6VuHYFIhH4LITTOeSBWwnxCCOE8M9sLSDezd2Mvbzu+JzDdzN4KIXxevCGL5E8tApE4CSH8CkwF6uxw/DcgnSIWBhOJihKBSMHtmaNb6K0dXzSzynhNo1k7HN8fr/czvnjCFCkcdQ2JFNxOXUMxp5jZdGAr0CdWAuG02PGv8STwRAjhx2KMVaTAlAhEim5CCOG8vI6bWW1gopkNDyGkF3dwIvlR15BIxGLloPsA3RIdi0hulAhEisdAoHlslpFIiaLqoyIiKU4tAhGRFKdEICKS4pQIRERSnBKBiEiKUyIQEUlxSgQiIilOiUBEJMX9P9eK22JIoz0hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle as cPickle\n",
    "\n",
    "cPickle.dump(grid.best_estimator_, open(dpath + \"LR_data/kkbox_LR_Saga.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
